09. Data Preparation

Error: The formula for the upper fence is shown incorrectly as Q3−1.5×IQR. It should be Q3+1.5×IQR.

Data Preparation Essentials for AI Models

Key Processes and Techniques:

• Data Cleaning:

  • Remove Duplicates:
  • Identical data rows, often due to errors, can lead to data leakage. Eliminate these to maintain data integrity.
  • Handle Missing Values:
  • Missing data can appear as Null, None, NA, NaN, or empty strings.
  • Drop Rows: Direct but potentially wasteful if valuable data is omitted.
  • Impute Missing Values: Use mean, median, or more advanced models like KNN.

• Outlier Management:

  • Outliers can be errors or rare valid events.
  • Identify using IQR: If a value falls outside 1.5 times the interquartile range, consider it an outlier.
  • Handling Techniques:
  • Trimming: Remove incorrect values.
  • Capping and Flooring: Adjust to less extreme figures.
  • Leave Intact: Sometimes necessary if outliers contain vital information.

Ongoing Process:

• Data preparation is iterative.
• Involve exploratory data analysis, feature engineering, and modeling.
• Fabricate the approach based on domain expertise for optimal results.

QUIZ QUESTION::

Match each data issue with its appropriate cleaning/handling strategies.

ANSWER CHOICES:

Conduct outlier analysis and use expert judgement

Expand dataset with external sources

Create new data row

Impute missing values

Standardize formats

Remove duplicates

Ignore inconsistencies

Data Issues	Cleaning/Handling Strategies
Multiple identical entries
Inconsistent or incorrect data entry (e.g., date and number formats)
Missing data in critically important columns
Extreme values that skew the data

SOLUTION:

Data Issues	Cleaning/Handling Strategies
Extreme values that skew the data	Conduct outlier analysis and use expert judgement
Missing data in critically important columns	Impute missing values
Inconsistent or incorrect data entry (e.g., date and number formats)	Standardize formats
Multiple identical entries	Remove duplicates